Government 
Publications 


3 1761 11556220 9 


Aboriginal DiabetesInitiative 


| 
| 


Evaluation 


Framework 


February 2002 


Hell CShade Canada | | Canada 


Digitized by the Internet Archive 
in 2022 with funding from 
University of Toronto 


https://archive.org/details/31/61115562209 


Aboriginal DiaberesInitiative 


Evaluation 


Framework 


February 2002 


Our mission is to help the people of Canada maintain and 


improve their health. 
Health Canada 


This Evaluation Framework was drafted by the Accountability and Evaluation 


Working Group of the Aboriginal Diabetes Initiative. The group consisted of: 


Dr. Gillian Bailey, First Nations and Inuit Health Branch (FNIHB), Health Canada 

Michelle Callaway, FNIHB, Health Canada 

Kidane Gebremariam, FNIHB, Health Canada 

Barbara LeBrun, Population and Public Health Branch, Health Canada 

Joanne Lucarz-Simpson, FNIHB, Health Canada 

Alastair MacPhee, Congress of Aboriginal Peoples 

Onalee Randell, Inuit Tapiriiksat Kanatami (ITK) 

Satish Seetharam, FNIHB, Health Canada 

Anita Stevens, Assembly of First Nations 

Susan Taylor-Clapp, FNIHB, Health Canada 

Maureen Thompson, FNIHB, Health Canada 

Krista Van Aert, FNIHB, Health Canada 

Francine Vincent, First Nations of Quebec and Labrador Health and 
Social Services Commission 

Joanette Wallace, FNIHB, Health Canada 

Joan Wentworth, Battleford Tribal Council 


For all enquires regarding this document, please contact 


Satish Seetharam 

Program Evaluation Manager 
Community Health Programs Directorate 
Jeanne Mance Bldg., Room 2007 
Tunney’s Pasture 

Ottawa, Ontario, K1A OL3 

Phone (613) 952-4230 

Fax (613) 954-8107 


Email: satish_seetharam@hc-sc.gc.ca 
Published by authority of the Minister of Health 


This publication can be made available in/on computer diskette/large 


print/audio cassette/braille upon request. 


© Minister of Public Works and Government Services, 2002 
Cat. H35-4/20-2002 
ISBN: 0-662-66385-3 


aan ®nal <i ee ai Solon «6G 


Table of Contents 


1.0 Purpose and Background Information .......... 1 
1.1. Purpose of the Aboriginal Diabetes Initiative Evaluation ..... 1 


1.2 Background and Description of the Aboriginal Diabetes 


IHALIVERS?, Sc oie. mM pet mene r enemy ies SARE aM a lets ai ins 4 
128s Prog ran ODjSCUVES x acs ny Aendlanl ah Myerdesta tou, Ryhen Jens mal eae apna 8 
Maes Program: Component Outcomes’... ts soe e eae ne 2 
TSS WADILEUINGINO LCV ClSeanc:c. cs 67 etter nek coy ayo ieee a ee eee 10 
Gis Silay Ger Gl Oup sien. ait he ee ee en 10 
Ze Programumplementation-and Delivery: 24) .cni ee weave 11 
(| Gee PLOgranielOGie MOdEl™. wt tte me rae eres ere 14 
Z:0,.EVallation Strategy = inves eee ree ies 
Dhl Pe EVallaviGn: @iallenGQe sam wieer oe Rie atte tien acm alee, Se NZ 
ZED eee Nall td all Oh GUS tues eae rata fre, Sines tte ome hen Oe Oye 20) 
2.3 Evaluation Questions and Indicators ................... 21 
2.4 Component-Level Evaluation Questions and Indicators ..... 23 
Zio a WMethCadologyiand Pata sOUNCeS.c Gf aa.ts eter. ree i ses 26 
3:0: sThesEvaluation a deters oes ts & Bee ane 31 
Sal NE Va UauON ECON Cer are coset We tiene ene cece athena eat yay 31 
3.2, sEValuatlon: RESPOMSIDINDY! 22a wuts ae oo myers taeenes exatsne ies 3] 


3.3 Changes to the ADI Evaluation Framework .............. BZ 


Aan pidts feral _ 


a ee - 
_ —— 

; ait ps a 

_ 
mar a, . oe 
_ = =~ : ; 
Wd nal 

j 

a ahs 

i‘ 
‘% Apartaiiia'’y % — 


tawievea wii 0.5 


Sau Ae 6G UY 


2.3) 180 aes and Background 


_ Information 


1.1. Purpose of the Aboriginal Diabetes Initiative 
Evaluation 


Overview of the Canadian Diabetes Strategy 


Diabetes is a chronic disease where the body fails to produce insulin 
(type 1) or cannot properly use insulin (type 2), a hormone essential 
for normal body metabolism. Approximately 90 percent of all diabetes 
cases among Canadians are type 2, which usually occurs after age | 
40. Seniors, Aboriginal people, baby boomers and other high-risk 
groups (those at greatest risk of weight gain, those with a sedentary 
lifestyle) are therefore key targets for the Canadian Diabetes Strategy 
(CDS). State of the art knowledge indicates that programs which 
address key modifiable risk factors of obesity and inactivity should 


help prevent/delay the onset of type 2 and its complications. 


Yearly, it is estimated that diabetes costs Canadians $9 billion, is 
directly responsible for 5,500 deaths and contributes to 25,000 
potential person years of life lost. Incidence increases by 60,000 new 
cases every year. It is estimated that over two million Canadians have 


diabetes, and an estimated one-third of the cases are undiagnosed. 


Diabetes is three to five times more prevalent in Aboriginal populations 
than in the general population. Among the Inuit, the rates of diabetes 
are lower than in the general population. Type 2 diabetes accounts 
for almost all cases of diabetes amongst Aboriginal people, and 
generally occurs at a much younger age: it has appeared in 
Aboriginal children as young as five. Long duration of high blood 
sugar levels is correlated with complications, hence there is great 
concern that this early onset of diabetes will lead to early onset of 


complications. 


The 1999 federal budget committed the federal government to 
the development of the Canadian Diabetes Strategy — a strategy 


of $115 million over five years, which includes a major Aboriginal 


component. (The CDS was formerly known as the Canadian Diabetes 


Prevention and Control Strategy, initially set at $55 million over three 
years). This commitment evolved from the 1997 Speech from the 
Throne that committed to addressing the rapid increase in diabetes 
in Aboriginal communities, the 1998 announcement of Gathering 
Strength — Canada’s Aboriginal Action Plan, and increasing pressure for 


a Canada-wide diabetes strategy from national groups. 


The Government of Canada is engaging all stakeholders in activities 
to address diabetes and its complications. In consultations with 
provincial and territorial governments, Aboriginal organizations, and 
national diabetes and nonprofit groups, consensus was reached on 
four strategic components and priority areas for action: Prevention 
and Promotion; Aboriginal Diabetes Initiative; National Diabetes 
Surveillance System; and National Coordination. These four elements 


recognize that: 
e type 2 diabetes is thought to be mainly preventable; 
e Aboriginal populations are particularly at risk; 


* agreater knowledge base is required to promote informed 


decision-making; and 


* pooling efforts, reaching consensus, complementing and 
building on existing resources will engender the greatest 


benefits. 


Liaison will continue throughout the duration of the Strategy. 
Advisory Committees will actively engage all stakeholders, where 
appropriate, to maximize the investment of the federal government 


in diabetes and also to jook toward the future. 


The overall objective of the CDS is to set the stage for significantly 
reducing the incidence and prevalence of diabetes and diabetes- 


related complications in Canada. The CDS will address the needs of 
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Canadians by focusing on assisting those populations which, 
according to state-of-the-art knowledge, are most at risk for 
developing diabetes and its complications: Aboriginal people, 
seniors, people who are overweight, people with sedentary 


lifestyles and Canadians already living with the disease. 


The CDS is premised on the recognition that all levels of govern- 
ment, as well as non-governmental organizations (NGOs), Aboriginal 
organizations, communities and individuals have important roles to 
play, and contributions to make in addressing diabetes in Canada. 
The CDS therefore seeks to develop and facilitate partnerships with 
all key stakeholders. The CDS design involves branches of Health 
Canada working together to set the stage for achievement of positive 
health outcomes that will contribute to the overall mandate and 
mission of Health Canada: To maintain and improve the health of 
Canadians. Consultations and ongoing dialogue with the provinces 
and the territories have singled out health promotion and type 2 
diabetes prevention as the actions likely to reap the greatest benefits 
in addressing diabetes, especially for those at risk of developing 


type 2 diabetes. 


Evaluation of the Aboriginal Diabetes Initiative 


The Aboriginal Diabetes Initiative (ADI) requires an ongoing monitoring 
and evaluation strategy to assist with the management of activities 
and provide ongoing information on the progress toward national 
goals. It will also provide information on intermediate impacts and 


outcomes resulting from implementation of the program. 


The Evaluation Framework is designed and developed for the formal 
evaluation of the Aboriginal Diabetes Initiative. This framework is 
designed to identify the specific components of the ADI as approved 
by Treasury Board. For each component, the long-term objectives 
and key indicators are presented. A Logic Model provides the major 
elements of the evaluation framework including what is to be evaluated 


and the expected activities and outputs. 
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The evaluation methodology described in this report identifies 


existing information sources, new (proposed) information collection 
mechanisms (consultations/key informant interviews, surveys, special 
studies, etc.) and the independent evaluations of the three ADI 
components, in order to evaluate the effectiveness of activities and 


projects in contributing to the common goals. 


The Evaluation Framework states what is to be evaluated (the objec- 
tives and activities/outputs of the three program components), what 
sources of information will be used, and what impacts and effects 
the activities demonstrate. The evaluation framework begins with 
the Aboriginal Diabetes Initiative, followed by each of the three 
components of the ADI: 1) care and treatment; 2) prevention 

and promotion; and, 3) lifestyle support. This framework report 

is developed to complement and expand upon those developed 

for the overall Canadian Diabetes Strategy and First Nations and 


Inuit Health Branch (FNIHB) evaluation frameworks. 


Background and Description of the Aboriginal 
Diabetes Initiative (ADI) 


General Background Information 


The ADI is a response to a critical need for action. It strives to create 
a holistic environment for good health, lifestyle changes, community 
linkages and partnerships. There is a strong need for collaboration 
and partnership on many levels, including care and treatment 
providers, community members, leaders, FNIHB regions, NGOs and 
others, in order to develop adequate and effective Care and 
Treatment, Prevention and Promotion, and Lifestyle Support 
programs and services, which are culturally appropriate and also 


community-based and delivered. 


Under the umbrella of the CDS, and in keeping with Health Canada’s 
unique responsibilities, the goal of the ADI is to begin to increase 


awareness of type 2 diabetes and reduce the incidence of its 


complications in Aboriginal people by implementing culturally 


sensitive, holistic, and accessible programs. The ADI will do this by 
providing access to direct care and treatment and lifestyle support 
programs and services for First Nations living on reserves and Inuit 
living in Inuit communities. It will also provide an overall culturally 
appropriate approach to prevention and health promotion 
programs for all Aboriginal peoples, including Métis, urban Inuit 


and off-reserve populations. 


The ADI is one of four components of the CDS, the Government 

of Canada’s response to Diabetes, which involves many stakeholders 

and partners. It also has a strong link with the First Nations and Inuit 
Home and Community Care (FNIHCC) Program. The ADI is a 5 year 

program, at the end of which Health Canada must report to Treasury 
Board. An on-going and outcome based evaluation is necessary for 


the ADI, in accordance with the Treasury Board submission. 


Program Description 


The ADI will begin to address the epidemic of diabetes among 


Aboriginal people by focusing its efforts in three main areas: 


e Care and Treatment; 
e Prevention and Promotion; and, 


e Lifestyle Support. 
The ADI is comprised of two programs: 


1) First Nations On-Reserve and Inuit in Inuit Communities 
Program (FNOIIC); and, 


2) Métis, Off-Reserve Aboriginal and Urban Inuit Prevention and 


Promotion Program (MOAUIPP). 


Details of these programs are presented in the two logic models 


on pages 14 and 15. 
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The ADI for First Nations On-Reserve and Inuit in Inuit Communities 
makes up 75% of the program, and it focuses on all three major 
program components (care and treatment, prevention and promo- 
tion, and lifestyle support). These three major components will be 


delivered: 


e through the FNIHB regional processes; 
e through the funding formula; 
* according to a general set of guidelines; and, 


e in partnership with communities. 


The Métis, Off-Reserve Aboriginal and Urban Inuit Prevention and 
Promotion Program (MOAUIPP) focuses on primary prevention and 


health promotion. It will: 


e be delivered through a headquarters request for proposal process; 
e be based on a funding formula; 
e follow program guidelines; and 


® encourage partnerships, wherever possible. 


Program Components 


1. Care and Treatment — In keeping with Health Canada’s unique 
responsibility to ensure access to direct care and treatment services 
to First Nations on-reserve and Inuit living in Inuit communities, the 
care and treatment element will be specific to these groups. (Provinces 
and territories will continue to provide care and treatment services to 


Métis, off-reserve Aboriginal and urban Inuit, as is their jurisdiction.) 


Care and Treatment will comprise one part of the ADI, and will be 
delivered through contribution agreements, administered through 
FNIHB regions. Programs (such as screening programs, foot and eye 
clinics, regular examinations, tests and treatment programs) will 
serve to meet the care and treatment needs of First Nations and Inuit 
already diagnosed with diabetes, and to prevent further complications 
from developing. Programs will be developed and delivered in a 
culturally sensitive and holistic manner by the communities them- 


selves. Activities will be community-based, to ensure that First Nations 


and Inuit are able to build the necessary capacity to manage sustainable 


and culturally appropriate diabetes programs. 


2. Prevention and Promotion — Prevention and promotion programs 
will target all Aboriginal people: Métis, on- and off-reserve First 
Nations and Inuit. Programs will be culturally appropriate, and 

will be administered through contribution agreements and other 
arrangements. Consultations have indicated that primary prevention 
is a major concern of the Inuit who do not want to see the extremely 
high rates of diabetes experienced by First Nations replicated in their 
communities. First Nations have also indicated the importance of 
ensuring that youth and those most at risk are educated on diabetes 


risk factors. 


National organizations such as the Assembly of First Nations (AFN), 
the Inuit Tapiriiksat Kanatami (ITK), the Native Women’s Association 
of Canada (NWAC), the Métis National Council (MNC), the Congress 
of Aboriginal Peoples (CAP), and the National Aboriginal Diabetes 
Association (NADA) are participating in implementation planning 
and providing input into program delivery to their constituencies. 
Prevention and promotion activities will include best practice sites, 
train the trainer programs, awareness programs, school-based 
programs (on-reserve and in Inuit communities) and an Aboriginal 


resource centre/clearinghouse. 


3. Lifestyle Support — Lifestyle support services will be provided 
to First Nations and Inuit communities to help those living with 
diabetes and their families to cope with its devastating effects. 
Lifestyle support services for those living with diabetes are an 
important element of the actions required to address diabetes 


at the community level. 


Community support groups and systems which help enhance 
quality of life will be established and programs that help support 
self-management by promoting positive lifestyle choices will 

be provided and funded. Programs will be implemented at a 


community level, and will involve people living with diabetes, 
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their families, community leaders and community centres. Links to 
NGOs such as the Canadian National Institute for the Blind (CNIB), 
NADA and the Canadian Diabetes Association (CDA) will be 


encouraged. 


Program Objectives 


Working in partnership with Aboriginal committees, Health Canada’s 
investments are intended to achieve the following objectives for the 


two ADI programs: 


First Nations On-Reserve and Inuit in Inuit 
Communities 


e Raise awareness of diabetes, its risk factors, and the value of 


healthy lifestyle practices. 


e Support the development of a culturally appropriate approach to 
care and treatment, diabetes prevention and health promotion 


programs, and lifestyle support programs. 


e Build capacity, linkages and infrastructure for all components of 


the ADI in First Nation and Inuit communities. 
° Promote effective self-management of diabetes. 


* Coordinate with other community-based programming, 
specifically the First Nations and Inuit Home and Community 


Care program. 
Métis, Off-Reserve Aboriginal and Urban Inuit: 
Prevention and Promotion 


e Raise awareness of diabetes, its risk factors, and the value of 


healthy lifestyle practices. 


e Promote Aboriginal/Inuit ownership of diabetes primary 


prevention and health promotion programs. 
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e Ensure the fair and equitable allocation of available resources 


among Métis, off-reserve Aboriginal people and urban Inuit. 


e Ensure that programming is delivered as equitably as possible 


across the country. 


* Promote innovative approaches to diabetes primary prevention 


and health promotion projects. 


1.4 Program Component Outcomes 


The most important program outcomes associated with the three 


program components are presented in the following table. 


ADI Program 


Components Outcomes 


Care and Treatment ° Improved access to care and treatment for First Nations and Inuit with diabetes 
° Stronger partnerships with First Nations and Inuit diabetes care providers 


e Increased capacity of First Nations and Inuit communities to manage effective 
diabetes programs 


* Improved clinical outcomes including decreased rates and severity of 
complications and hospitalization 


G 


Prevention and Promotion |° Overall improvement in diabetes awareness: risk factors, signs and symptoms, 
benefits of healthy lifestyle choices 


° Implementation of effective diabetes prevention programs and projects 


° Production and distribution of resource materials appropriate to First Nations, 
Métis and Inuit cultures and languages 


e Improved early detection and reporting of diabetes 


Lifestyle Support e Effective community-level programs in place to provide support and services 
to people with diabetes, their families and care providers 

° Strengthened community-based organizations to support diabetes programs 
and services 

® Improvement in quality of life and self-sufficiency for First Nations and Inuit 
with diabetes 


eo 
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ADI Funding Levels 


The following table presents the five-year total and annual funding 


levels for the Aboriginal Diabetes Initiative. 


Amount 


(in Millions) 


1999-00 Z 
2000-01 11 
2001-02 15 
2002-03 15 
2003-04 1S 
Total 58 


Target Groups 


The ADI aims to increase awareness of diabetes, and reduce the 


incidence and prevalence of diabetes related complications among 


Aboriginal people including the following groups: 


First Nations and Inuit of any age: 


Those who live on a First Nations reserve, or in an Inuit 
community, or in a First Nations community in the Yukon, 


Northwest Territories or Nunavut; 
Métis living in Métis communities, rural or urban centres; 


First Nations people living off-reserve in either rural or urban 


centres; 
Inuit living outside Inuit communities or in urban areas; and, 


Non-Status Indians. 
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1.7. Program Implementation and Delivery 


The ADI is a five-year process. A National ADI Steering Committee 
was formed to guide the implementation and delivery of the ADI, 
and includes representatives from the major Aboriginal representative 
organizations, Health Canada, and the National Aboriginal Diabetes 


Association. 


° In year one of the ADI (1999/2000), the Aboriginal Diabetes 
Initiative Steering Committee held implementation planning 
meetings with representatives from Aboriginal organizations, 
communities, regional and headquarters staff early in the winter 
of 1999/2000. These national and regional meetings confirmed 
the program needs and priorities, identified opportunities, and 
led to a full implementation framework. Additionally, funding 
was provided to existing pilot projects that are already estab- 
lished (that is, projects previously funded by National Health 
and Research Development Program (NHRDP), and given bridge 
funding by Medical Services Branch (MSB), Health Canada in 
1998 to prevent the loss of capacity and momentum), to the 
National Aboriginal Diabetes Association, and to regionally-based 


implementation planning. 


e Year two of the ADI (2000/2001) focused on implementing the 
full range of care and treatment, prevention and promotion and 
lifestyle support programs on-reserve and in Inuit communities 
across the country through the FNOIIC process; and creating a 
Request for Proposal process which resulted in the implemen- 
tation of 16 primary prevention and health promotion programs 
under the MOAUIPP program. 


e Inyears three and four (2001/2002 and 2002/2003), many 
programs created in year two will be continued, complete 
with accountability and reporting frameworks. Additional 
programming will be added as the funding level increases to 


its maximum. 
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° In year five (2003/2004), programs with accountability 


and reporting frameworks will be continued. Analysis of the 
gaps in programming, and ongoing program needs will be 
undertaken and an evaluation of outcomes (immediate 


outcomes) will also be conducted. 


There will be two levels of ADI evaluations. Level | focuses on ongoing 
performance measurement. It will include monitoring, gap analysis 
and project evaluation for projects conducted in years three through 
five (2001/2002 through 2003/2004). Level II will be a final summa- 
tive evaluation focusing on final program outcomes and impacts and 
the effectiveness of the ADI in reaching program goals and objectives. 
The results of the summative evaluation will be reported to the 
Minister of Health. 


There is a need to build linkages with the Provinces, Territories, 
Regions, Communities, and other stakeholders. The ADI also has 

a linkage with the FNIHCC program, since much of the direct 
diabetes care and treatment services to those First Nations and Inuit 
living with diabetes will be delivered by FNIHCC workers in the 
community. There are also linkages with other programs such as 
Aboriginal Head Start (AHS), the Canada Prenatal Nutrition Program 
(CPNP), and the First Nations and Inuit Health Information 

System (FNIHIS). 
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ADI Evaluation Process 


Programs Programs Programs Programs 
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| 1999-2004 


| Components 


Activities 


Outputs 


Immediate 
Outcomes 


2004-2009 


Intermediate 
Outcomes 


| 2009 & after 


| Long-term 
Outcomes 
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Program Logic Model 


Aboriginal Diabetes Initiative 


i First Nations On-Reserve & Inuit in Inuit i aR 


Care and Treatment 


Prevention and Promotion 


Lifestyle Support 


y 
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e Establish linkages (external and existing 
programs) 


e Develop culturally-sensitive, community- 
based programs and services for screening 
and treatment 


e Provide care and treatment programs 


* Develop access to care and treatment 
services 


« Hire diabetes co-ordinators 


| « Support Diabetes Advisory Committees 


* Support training to health care workers 


e Establish linkages (external and existing 
programs) 


e Raise awareness of diabetes, its risk 
factors, and the value of healthy lifestyle 
practices 


e Provide culturally-appropriate programs 


* Promote access to culturally-appropriate 
programs 


e Promote First Nations and Inuit ownership 
of prevention programs 


* Promote effective management of 


e Establish linkages (external and existing 
programs) 

Establish community support groups and 
services to enhance quality of life for 
people with diabetes and their families 
and care providers 


* Develop programs and services to increase 
self-sufficiency and promote positive 
lifestyle choices 

Activities such as peer support groups, 
sharing circles, drop-in programs and 
recreational programs for youth at risk 
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diabetes 


Y 


* Care and treatment programs 
* Screening programs 


* Education programs 
(e.g., self-management) 

» Mechanisms to improve access to 
programs 

* Formal linkages and partnerships with 
First Nations and Inuit program providers 
(e.g. Home and Community Care) 


e Programs to raise awareness of diabetes 
problems and prevention 

e Education programs for youth and higher- 
risk groups 

° Healthy lifestyle and active living 
promotion programs 

* Community projects that strengthen local 
capacity 

* Diabetes resources: e.g. NADA newsletter 


¢ Community projects and services for 
people with diabetes and their families 
and care providers 


* Support programs to increase self- 
sufficiency 


° Support programs that promote healthy 
lifestyles 


Y 


Y 


Y 


* Increased community and organizational 
capacity to deliver diabetes programs and 
services for screening, care and treatment, 
and training 

* Improved coordination of actions on 
diabetes 


* Improved access to direct care and 
treatment programs on reserves and in 
Inuit communities 


* Initial community mobilization on 
diabetes 

« Improved coordination of actions on 
diabetes 

* Increased capacity of First Nations and 
Inuit organizations and communities to 
address diabetes issues 

* Resource materials appropriate to 
Aboriginal and Inuit cultures and 


e Increased community services for people 
with diabetes, families and care providers 


* Increase in community-based 
organizations that support diabetes 
programs and services 
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languages 


Y 


* Improved capacity to manage sustainable 
programs 

* Increased proportion of on-reserve and 
Inuit community populations served by 
care and treatment programs 


* Increase in the number of diabetes health 
care providers/workers 


* Improved care and treatment services 


* Higher levels of awareness of the health 
problems and complications associated 
with diabetes — warning signs and risk 
factors 


* Higher levels of awareness of prevention 
and benefits of healthy lifestyle choices 


Increased reporting of diabetes cases 


° Improvements in access to support for 
people with diabetes 


* Improvements in community-level 
support provided to people with diabetes, 
their families and care providers 


Y 


Y 


Y 


* Improved care and treatment services 


* Improvements in clinical outcomes; 
reduced mortality and complications 


* Improved screening and early detection of 
diabetes 


* Clinical practice guidelines that reflect 
circumstances of First Nations on-reserve 
and Inuit communities 


* Decreased rates and severity of 
complications and hospitalizations 


* Improved access to resource materials 
appropriate to First Nations and Inuit 
cultures and languages 


° Increased awareness 
* Reduced incidence of diabetes 


° Improvements in quality of life and self- 
sufficiency for First Nations and Inuit with 
diabetes 


* Ongoing lifestyle support services to First 
Nations and Inuit 
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Aboriginal Diabetes Initiative 


Métis, Off-Reserve Aboriginal & Urban Inuit 
Prevention & Promotion (MOAUIPP) 


Prevention and Promotion 


Establish linkages (external and existing programs) 
Raise awareness of diabetes, its risk factors, and the value of healthy lifestyle practices 
Provide culturally-appropriate programs 
Promote access to culturally-appropriate programs 


Promote Métis, off-reserve Aboriginal and Inuit ownership of prevention programs 


Diabetes awareness and prevention programs 
Education programs for youth and higher-risk groups 
Healthy lifestyle and active living promotion programs 


Community projects that strengthen local capacity 


Diabetes resources: e.g. NADA newsletter 


Community involvement in prevention and promotion programs 
Improved coordination of actions on diabetes 


Increased capacity of Métis, off-reserve Aboriginal and Inuit organizations and communities 
to address diabetes issues 


Access to diabetes prevention and promotion programs 


1999-2004 
Components 
Activities 
Outputs 
Immediate 
Outcomes 
2004-2009 


Intermediate 
Outcomes 


Higher levels of awareness of the health problems and complications associated with 
diabetes, warning signs, and risk factors 


Resource materials appropriate to Métis, off-reserve Aboriginal and Inuit cultures 
and languages 


Higher levels of awareness of prevention and benefits of healthy lifestyle choices 


Increased reporting of diabetes cases 


2009 and after 


Long-term 
Outcomes 


Improved access to resource materials appropriate to Métis, off-reserve Aboriginal 
and Inuit cultures and languages 


Increased awareness | 


Reduced incidence of diabetes 
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2.0 Evaluation Strategy 


Program evaluation is a tool for good management practices that is 
used on an ongoing basis. In most cases, program evaluation is done 
at the end of a program’s life cycle or according to a predetermined 
time frame. However, evaluation concepts should be incorporated on 
an ongoing basis at all stages of program delivery, especially at the 
front end, to articulate linkages between activities and outcomes. 
Though final evaluation of various components may be conducted, 

it is extremely important that timely, comprehensive, and strategic 
evaluation studies of projects and activities within each component 
are conducted on an ongoing basis to feed into the final evaluations 


of the components and the overall initiative. 


As presented in Section 1, there will be two levels of evaluation. 

A process evaluation will focus on ongoing program monitoring 
(Level 1). It will include monitoring, gap analysis and project 
evaluation for the three years of full program implementation. 
Process evaluation is on-going, and complements and strengthens 
the summative (outcome) evaluation. An outcome/summative 
evaluation (Level 2) will focus on final program outcomes and 
impacts and the effectiveness of the ADI in reaching program 


goals and objectives. 


Activities during the first year of bridge financing (1999-2000) will be 
covered during the first year of ongoing performance measurement 
(2000-2001). 


2.1 Evaluation Challenges 
The ADI Evaluation is an ongoing process covering a five-year time 
period and involving many stakeholders and Aboriginal organizations 


and communities from across the country. Following are some of the 


challenges to a successful ADI evaluation. 
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e Multiple players — Health Canada is only one of many players 


whose activities and policies influence the health of Canadians. 


e Limited mandate and resources — Health Canada has a limited 
mandate and resources to act on a wide spectrum of activities 


related to diabetes prevention, management and control. 


e Long-term co-ordination of efforts — Reaching these targets and 
objectives requires long-term, co-ordinated efforts from all 


players across all jurisdictions in Canada. 


e Limited time to measure ADI objectives — The time period of ADI, 
from 1999-2000 to 2003-2004, will not be sufficient to demon- 
strate the overall objectives. The evaluation will need to address 
the progress made by ADI as well as shorter-term or immediate 


outcomes. 


e Provincial and territorial health reforms — Provincial and territorial 
health reform initiatives will have an impact on the diabetes care 


and treatment of Aboriginal people. 


° — Jurisdictional issues for program delivery — The policies and programs 
of other federal departments, provincial and territorial govern- 
ments, and Aboriginal structures will have a direct impact on the 
health status of Aboriginal people. This issue may be particularly 
important in Nunavut and the Northwest Territories where there is 


no program delivery directly to Inuit except through government. 
e Determinants of health — There are a variety of factors, known 


as the determinants of health, which influence the health of 


Canadians. 
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e Establishing causality — In most instances, given the variety 
of factors and agents influencing change, it will be difficult to 
establish direct causal linkages between the ADI and changes 


in the health status of Aboriginal Canadians. 


e Multiple jurisdictions — There are many jurisdictions involved in 
the delivery of health care including provinces, territories, Health 
Canada, Aboriginal structures and non-governmental agencies. 
Achieving key results will require long-term efforts from all 
partners across all jurisdictions (local, regional and national), 


going beyond a five-year period. 


e Linkages with other First Nations and Inuit Home and community 
programs — ADI provides funding for specific types of projects. 
At any given point in time, many community programs are 
administered simultaneously: e.g. FNIHCC and AHS. In many 
cases, the programs are delivered collaboratively. Success or failure 
of the ADI could depend on how other programs are delivered 


in the community. 


e ADI and FNIHCC have different funding levels — In some commu- 
nities, ADI and FNIHCC resources are integrated to achieve better 
results. ADI provides funding for a defined period of time under 
its S year mandate while the FNIHCC program has ongoing 
funding. This difference could complicate the integration of 


activities and the development of delivery mechanisms. 


e — Tracking when funding is distributed — Many of the ADI projects 
will be sponsored and delivered at several levels including national, 
regional, Tribal Council and community levels. Tracking the 
timing of the funding distribution by organizations operating 
under different financial reporting practices will require reliable 


reporting mechanisms and ongoing monitoring. 
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e Sharing information among program partners and stakeholders — 


Sharing, retrieving and reporting on information from provinces, 
regions, communities and other stakeholders may be difficult. 
The large number of program participants will complicate 
communications. Managing the overlap and linkages of 
information from programs with different agendas and 
mandates, such as the FNIHCC program and The National 
Diabetes Surveillance System (NDSS), will present significant 
challenges. There may also be reluctance among delivery 
partners to establish new and additional communications 


protocols when existing ones are working properly. 


Evaluation Tiers 


The evaluation will be conducted at four tiers or levels of 


aggregation. 


* Communities; 
e Band Council/Tribal Council; 
e FNIHB Regional; and, 


e National. 


Evaluation data can be summarized to support analysis for each of 
these four tiers. For a program of the size and complexity of the ADI, 
specific data sources will be more useful for different evaluation tiers. 
For example, the NDSS will provide very useful data at the national 
and regional levels but the NDSS data may not support analysis at 
the Band/Tribal Council and community levels of analysis. On the 
other hand, consultations, reviews or case studies conducted locally 
may provide valuable results for community level evaluation but be 


insufficient to support analysis at the regional or national levels. 
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The tables presented in Section 2.3 and 2.4 present a preliminary 
review of the correspondence between different evaluation questions 
and methods and the four evaluation tiers. These relationships 

will have to be revisited after the evaluation methodology has 

been designed and the evaluator has had a chance to assess the 


quality of the data collected through the different methods. 


2.3. Evaluation Questions and Indicators 


Consistent with the approach of evaluating the CDS presented in the 
Treasury Board submission (CDS Evaluation Framework), the progress 
toward meeting the goals of the ADI will be measured according to 


the following criteria: 
e Relevance — whether program components are still needed; 


e Progress/Success — the extent to which activities and outputs 


have led to the specific outcomes identified in the program plan; 


e Implementation & Management - the extent to which 
implementation and management were appropriate and enabled 


the achievement of the identified outcomes; and, 


¢ Cost-effectiveness — the costs associated with each outcome. 


The general ADI evaluation questions that correspond to these four 
criteria, along with some key indicators, are presented in the 

following table. Preliminary lists of data sources and data collection 
methodologies as well as evaluation tiers are also presented. Similar 


tables for the three program components follow in Section 2.4. 
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Program-Level Evaluation Questions 


Key Evaluation Methodology/ Evaluation 
Questions Indicators Data Sources Tiers 
1. Relevance Incidence and prevalence NDSS e National 
° To what extent has the ADI rates of diabetes and its First Nations and Inuit e Regional 
been relevant to increasing complications Health Information System 
diabetes awareness, and Morbidity and mortality (FNIHIS) 
reducing the incidence and rates among people living First Nations, Métis and 
prevalence of diabetes-related with diabetes Inuit community 
complications among First organizations and care 
Nations, Métis and Inuit providers 
populations in Canada? 
ea Progress/Success Improved clinical NDSS/ FNIHIS e National 
° To what extent have ADI outcomes Aboriginal community ° Regional 
activities contributed to Higher rates of awareness organizations and care © Band Council/ 
outcomes? of diabetes risk factors and providers Tribal Council 
° Improved access to care and prevention Surveys of Aboriginal ° Community 
treatment Improvements in quality of populations 
e Increased awareness and Ne and self-sufficiency for Consultations with people 
prevention of diabetes First Nations and Inuit with living with diabetes 
: diabetes 
* Community programs to 
| improve quality of life and 
| self-sufficiency for people 
| with diabetes 
| 3. Implementation and Program/project funding Program data review ° National 
Management reflects the ADI objectives Case study/review of °* Regional 
° To what extent are the and priorities selected projects 
| implementation and Funding allocation Consultations with 
management of the ADI according to program plan program managers, 
eRe Opiate to acuievertie Collaborative arrangements project sponsors, and 
program objectives? with Aboriginal care Aboriginal organizations 
providers 
| 4. Cost-effectiveness Overall ADI cost- Program administrative * National 
° To what extent has the ADI effectiveness data review ° Regional 
used the most appropriate, Cost-effectiveness of Case study/review of 
efficient and cost-effective different program options selected projects 
methods to achieve program and project types 


objectives? 


Appropriateness of 
diabetes initiatives to 
Aboriginal peoples and 
communities 


Cost-benefit analysis of 
ADI project types 
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2.4 Component-Level Evaluation Questions and Indicators 


Care and Treatment 


Key Evaluation 


Questions 


Indicators 


Methodology/ 
Data Sources 


Evaluation 
Tiers 


Has ADI improved access Increased number of on-reserve e NDSS/FNIHIS National 
to care and treatment for and Inuit communities served by © First Nations and Inuit Regional 
First ae and Inuit with diabetes care and treatment community organizations Bend corned’ 
diabetes: raced and care providers Trial Conall 
Increased proportions of on-reserve 
: ; ; Community 
and Inuit populations with access 
to care and treatment 
Improved screening and early 
detection of diabetes 
Has ADI increased the Stronger partnerships with First e Consultations with National 
capacity of First Nations Nations and Inuit organizations Aboriginal organizations Regional 


and Inuit communities to 
manage effective and 
sustainable diabetes 


providing care and treatment 


Greater participation of First 
Nations and Inuit organizations in 


and care providers 


Consultations with 
program managers, health 


Band Council/ 
Tribal Council 


‘ Communit 
programs? care and treatment programs care providers y 
Improved capacity to manage e Review of project files 
diabetes programs 
Did ADI promote Partnerships in delivery of e Program data review National 
effective management of programs e Consultations with Regional 
diabetes? Timely delivery of programs and program managers, 
Me y of prog Band Council/ 
services project sponsors, and Tribal Council 
Aboriginal organizations @onamunity 
Has the ADI built capacity, Increase in Aboriginal e Program data review National 
linkages and infrastructure organizations capable of delivering | « Consultations with Regional 
for all components of the diabetes programs and services program managers, Band Council/ 
ADI in First Nation and Partnerships in program delivery project sponsors, and Tribal Council 
Inuit communities? Aboriginal organizations 
Community 
e Provincial and Territorial 
governments 
Has the ADI coordinated Partnerships in delivery of e Program data review National 
with other community- programs e Consultations with ADI Regional 


based programming, 

_ specifically the First 
Nations and Inuit Home 
and Community Care 
program? 


and other program 
managers, project 
sponsors, and Aboriginal 
organizations 
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Prevention and Promotion 


Indicators 


Methodology/ 
Data Sources 


Evaluation 
Tiers 


effective diabetes 
prevention programs for 


programs and projects 
Increased awareness of diabetes, 


Consultations with 
Aboriginal community 


Has ADI led to an overall Increased awareness among NDSS/FNIHIS National 
improvement in awareness community organizations Consultations with First Regional 
of diabetes, its risk factors, Increased awareness among First Nations, Métis and Inuit Community 
and the value of healthy Nations, Métis and Inuit community organizations 
_ lifestyle practices? populations and care providers 
Awareness among high-risk Surveys of Aboriginal 
subgroups populations 
| Has ADI provided for Implementation of diabetes NDSS/FNIHIS National 
implementation of awareness and prevention Program data review Regional 


Band Council/ 
Tribal Council 


all Aboriginal people? risk factors, and benefits of healthy organizations and care Community 
lifestyle practices providers 
Reduced incidence of diabetes Consultations with 
program managers 
Has ADI supported the Support of First Nations, Métis Review of program data National 
development of a and Inuit organizations Case study/Review of Regional 


culturally appropriate 


| approach to prevention 


Development and use of resource 
materials appropriate to 


selected projects 
Consultations with project 


Band Council/ 
Tribal Council 


diabetes primary 


| prevention and health 


promotion projects? 


First Nations, Métis and Inuit 
people 


selected projects 
Consultations with 
program managers, 
project sponsors and 
Aboriginal organizations 


| and health promotion Aboriginal languages and cultures sponsors and Aboriginal Community 
| programs? organizations 
| Has ADI promoted Partnerships in delivery of Review of program data National 
Aboriginal/Inuit ownership programs Case study/Review of Regional 
of diabetes primary First Nations, Métis and Inuit selected projects 
prevention and health sponsorship of ADI projects Consultations with project 
| promotion programs? sponsors and Aboriginal 
organizations 
Has the allocation of Percentage allocation of program Review of program data National 
available ADI resources funds among target groups Regional 
been fair and equitable 
among Metis, off-reserve 
| Aboriginal people and 
urban Inuit? 
Has ADI promoted Peer ratings of ADI projects Review of program data National 
innovative approaches to Introduction of new programs for Case study/Review of Regional 


Band Council/ 
Tribal Council 
Community 
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Key Evaluation Methodology/ Evaluation 
Questions Indicators Data Sources Tiers 
Have effective and relevant Timely implementation of ADI- Review of program data National 
diabetes support programs sponsored projects Consultations with Regional 
SARS been put in Peer ratings of effectiveness and program managers, Band Council/ 
piace: relevance of programs and services project sponsors, Tribal Council 
introduced Aboriginal and Inuit 
Pale: Community 
organizations 
Is there an improvement in Increased access to support NDSS/FNIHIS National 
the quality of life for First services for individuals with Consultations with Regional 
Nations and Inuit with diabetes, their families and care cai : 
diabetes because of ADI? roviders Rene aaa eee 
e ; organizations Tribal Council 
Support cM es created in Survey and consultations Community 
CORRS with individuals with 
diabetes and care providers 
To what extent has ADI Increased awareness of diabetes Review of program data National 
promoted self-sufficiency management practices Consultations with Regional 


and management of 
diabetes for First Nations 
and Inuit? 


Increased access to support 
services 


Increased availability and access 
to equipment and supplies 
needed for self-management 


Aboriginal and Inuit 
organizations 


Survey and consultations 
with individuals with 
diabetes and care 
providers 
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2.5 Methodology and Data Sources 


Wherever possible, the methods and data sources used to 

evaluate the ADI will be similar to those used for the evaluation of 
the overall CDS. These include the NDSS, the primary data source for 
monitoring and evaluating the CDS. Other sources developed for the 
CDS that could be used for the ADI evaluation (perhaps in modified 
form) include the template for minimum data collection from funded 


community projects, and the MOAUIPP project database. 


Following are some of the other data sources and methods that could 
be used for the ADI evaluation. This list is preliminary and presents 
possible sources only. A more precise specification of data sources 
and methods will be made during the evaluation design when more 


information about ADI projects and participants is available. 


A. Data Collection Methodologies 


Administrative program data review — The large array of data 
collecting during the process of administering the program will be 
an important source of information for the evaluation. Some of these 


data sources are as follows: 


e ADI project data including applications, workplans, and progress 


reports (where required and/or available); 
® activity reports; 
e funding and financial reports; 


® reports from related programs such as the NDSS and FNIHCC; 


and, 
* annual reports from CDS. 
Project document review — Every project will have some documen- 


tation in addition to the initial application. Such documentation 


could include communications materials (e.g. brochures, posters, 
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videos), training guides, planning documents, meeting agendas 

or notes, correspondence, and summaries or reports of various 
types. Asking sponsors to provide key documents produced during 
their project activities will assist Health Canada with ADI monitoring 
and evaluation. Sampling projects and requesting more detailed 
documentation about the activities and outcomes can also provide 


valuable information for a project review. 


Key informant interviews — Consultations with stakeholders, 
program partners, project sponsors and other relevant individuals 
and organizations can be conducted through key informant 
interviews. Key informant interviews are usually conducted with a 
single individual and are confidential. They can also be conducted 
with small groups. Both in-person and telephone interviews can be 
used depending on the subject and length of time required for 
the interview and the preferences of the respondent. Practical 


considerations such as travel costs are also very important. 


Expert Interviews — Consultations with several types of experts may 
be useful to program monitoring and evaluation. Medical personnel 
with expertise in diabetes, community care experts, social workers 
and counsellors, nutrition and recreation experts, and communications 
and marketing experts could make useful contributions to the evalu- 
ation. An understanding of and sensitivity to the First Nations, Métis 
and Inuit populations served by the ADI would be essential for 


experts consulted for the evaluation. 


Surveys of populations served by ADI — Surveys of larger numbers 

of program clients (i.e. representative probability samples) can help 
to provide a broad overview of program outcomes and impacts. 
Such surveys typically collect relatively small amounts of quantitative 
information from large numbers of people. The advantage is that 
the information collected can be used to make statistical inferences 
(within identifiable error parameters) to the Aboriginal populations 


served by the program. Surveys of Aboriginal populations present 


ra 
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particular challenges in terms of interview methods, access to 


representative samples of study populations (sample bias), and 
logistics. Careful planning and the support of the groups being 


surveyed are essential to success. 


Project reviews and evaluations — Some organizations sponsoring ADI 
projects may have their own processes for reviewing or evaluating 
activities. Methods could include case studies, self-evaluations, and 
community consultations. Such information could be incorporated 
into ADI monitoring and evaluation where organizations are willing 


to share their results. 


Case studies/reviews of projects — Case studies or other types of 
more detailed reviews could be conducted for selected projects. 
Case studies, which typically involve between two and five days of a 
researcher’s time, would involve in-depth examinations of all aspects 
of particular projects. Case studies would provide detailed evidence 
about the activities of a project and its outcomes and impacts. A 
good case study will also provide insights into the reasons for 


success or failure. 


B. Methods of Analysis 


Performance monitoring — Performance monitoring can be conducted 
for ADI projects for which progress data is available. Progress data 
collection should include measures that correspond to the indicators 


presented in this evaluation framework. 


Cost-benefit analysis — At the project level, cost-benefit analysis 
would require detailed data that could be collected through a case 
study or project audit. Cost-benefit analysis can be conducted after 
sufficient time has passed to allow for meaningful measurement of 
outcomes and benefits. Cost-benefit analysis at higher levels of 
aggregation could be conducted once adequate project-level data 


is available. 
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Analysis of linkages — An analysis of linkages with other key 
programs could be conducted at an early stage in the process. Key 
stakeholders should be consulted for an early review of the extent to 


which ADI is being integrated successfully with other programs. 


Gap analysis — A gap analysis can be conducted to make sure that 
workplans and activities conform to ADI objectives. Gap analysis 
involves the systematic comparison between planned activities 

and actual with the stated objectives of the ADI. These comparisons 
can be made at each level of analysis. Gap analysis can be used to 
make sure that the program is on track and to identify any required 
mid-program changes. 
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3.0 TheEvaluation 


This section covers the costing for the ADI evaluation, identifies 
responsibilities, and provides an opportunity for changes to this 


Evaluation Framework. 


3.1. Evaluation Funding 


Evaluation funding will be at two levels, with funding commensurate 
to the areas of responsibility. National Office will provide technical 
and financial support to the Regions, with the amounts to be . 


discussed and reviewed on an ongoing basis. 


3.2 Evaluation Responsibility 


The Community Health Programs Directorate of the First Nations 
and Inuit Health Branch of Health Canada will be responsible for the 
ADI evaluation, including the ongoing performance measurement 


over the next three years and the final summative evaluation in 2004. 


National Office will be responsible for the evaluation framework and 
design, and technical and financial support to Regions (as required). 
Regions will be responsible for regional level projects and establishing 


connections with Tribal Council and community level projects. 


Evaluation Responsibilities 


ADI Evaluation Framework National Office 
Gap Analysis National Office 
CDS Annual Reports National Office 
ADI Annual Reports National Office 
Evaluation — Ongoing National Office, 
Performance Measurement Regional Offices 
Evaluation — Final Summative National Office, 
Evaluation Study Regional Offices 


31 


3.3 


a2 


Changes to the ADI Evaluation Framework 


This evaluation framework, prepared in July 2001 (reviewed and 
edited in November), reflects the design of the ADI and incorporates 
the two programs and three components. It is likely that further 
changes in the upcoming years could affect the contents of this 
Framework. The Director General, Community Health Programs, 

will be responsible for accommodating those changes and for 


notifying all stakeholders including Treasury Board. 
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